Reliability and validity in comparative studies of software prediction models

Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest that it is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models

Missing Slice Imputation in Population CMR Imaging via Conditional Generative Adversarial Nets

Accurate ventricular volume measurements depend on complete heart coverage in cardiac magnetic resonance (CMR) from where most immediate indicators of normal/abnormal cardiac function are available non-invasively. However, incomplete coverage, especially missing basal or apical slices in CMR sequences is insufficiently addressed in population imaging and current clinical research studies yet has important impact on volume calculation accuracy. In this work, we propose a new deep architecture, coined Missing Slice Imputation Generative Adversarial Network (MSIGAN), to learn key features of cardiac short-axis (SAX) slices across different positions, and use them as conditional variables to effectively infer missing slices in the query volumes. In MSIGAN, the slices are first mapped to latent vectors with position features through a regression net. The latent vector corresponding to the desired position is then projected onto the slice manifold conditional on slice intensity through a generator net. The latent vector along with the slice features (i.e., intensity) and desired position control the generation vs. regression. Two adversarial networks are imposed on the regressor and generator, encouraging more realistic slices. Experimental results show that our method outperforms the previous state-of-the-art in missing slice imputation for cardiac MRI

Fuzzy min-max neural networks for categorical data: application to missing data imputation

The fuzzy min–max neural network classifier is a supervised learning method. This classifier takes the hybrid neural networks and fuzzy systems approach. All input variables in the network are required to correspond to continuously valued variables, and this can be a significant constraint in many real-world situations where there are not only quantitative but also categorical data. The usual way of dealing with this type of variables is to replace the categorical by numerical values and treat them as if they were continuously valued. But this method, implicitly defines a possibly unsuitable metric for the categories. A number of different procedures have been proposed to tackle the problem. In this article, we present a new method. The procedure extends the fuzzy min–max neural network input to categorical variables by introducing new fuzzy sets, a new operation, and a new architecture. This provides for greater flexibility and wider application. The proposed method is then applied to missing data imputation in voting intention polls. The micro data—the set of the respondents’ individual answers to the questions—of this type of poll are especially suited for evaluating the method since they include a large number of numerical and categorical attributes

Using Real-World Data in Health Technology Assessment (HTA) Practice:A Comparative Study of Five HTA Agencies

BACKGROUND: Reimbursement decisions are conventionally based on evidence from randomised controlled trials (RCTs), which often have high internal validity but low external validity. Real-world data (RWD) may provide complimentary evidence for relative effectiveness assessments (REAs) and cost-effectiveness assessments (CEAs). This study examines whether RWD is incorporated in health technology assessment (HTA) of melanoma drugs by European HTA agencies, as well as differences in RWD use between agencies and across time. METHODS: HTA reports published between 1 January 2011 and 31 December 2016 were retrieved from websites of agencies representing five jurisdictions: England [National Institute for Health and Care Excellence (NICE)], Scotland [Scottish Medicines Consortium (SMC)], France [Haute Autorité de santé (HAS)], Germany [Institute for Quality and Efficacy in Healthcare (IQWiG)] and The Netherlands [Zorginstituut Nederland (ZIN)]. A standardized data extraction form was used to extract information on RWD inclusion for both REAs and CEAs. RESULTS: Overall, 52 reports were retrieved, all of which contained REAs; CEAs were present in 25 of the reports. RWD was included in 28 of the 52 REAs (54%), mainly to estimate melanoma prevalence, and in 22 of the 25 (88%) CEAs, mainly to extrapolate long-term effectiveness and/or identify drug-related costs. Differences emerged between agencies regarding RWD use in REAs; the ZIN and IQWiG cited RWD for evidence on prevalence, whereas the NICE, SMC and HAS additionally cited RWD use for drug effectiveness. No visible trend for RWD use in REAs and CEAs over time was observed. CONCLUSION: In general, RWD inclusion was higher in CEAs than REAs, and was mostly used to estimate melanoma prevalence in REAs or to predict long-term effectiveness in CEAs. Differences emerged between agencies' use of RWD; however, no visible trends for RWD use over time were observed

Pain, pain intensity and pain disability in high school students are differently associated with physical activity, screening hours and sleep

Background: Studies exploring the association between physical activity, screen time and sleep and pain usually focus on a limited number of painful body sites. Nevertheless, pain at different body sites is likely to be of different nature. Therefore, this study aims to explore and compare the association between time spent in self-reported physical activity, in screen based activities and sleeping and i) pain presence in the last 7-days for 9 different body sites; ii) pain intensity at 9 different body sites and iii) global disability. Methods: Nine hundred sixty nine students completed a questionnaire on pain, time spent in moderate and vigorous physical activity, screen based time watching TV/DVD, playing, using mobile phones and computers and sleeping hours. Univariate and multivariate associations between pain presence, pain intensity and disability and physical activity, screen based time and sleeping hours were investigated. Results: Pain presence: sleeping remained in the multivariable model for the neck, mid back, wrists, knees and ankles/feet (OR 1.17 to 2.11); moderate physical activity remained in the multivariate model for the neck, shoulders, wrists, hips and ankles/feet (OR 1.06 to 1.08); vigorous physical activity remained in the multivariate model for mid back, knees and ankles/feet (OR 1.05 to 1.09) and screen time remained in the multivariate model for the low back (OR = 2.34. Pain intensity: screen time and moderate physical activity remained in the multivariable model for pain intensity at the neck, mid back, low back, shoulder, knees and ankles/feet (Rp2 0.02 to 0.04) and at the wrists (Rp2 = 0.04), respectively. Disability showed no association with sleeping, screen time or physical activity. Conclusions: This study suggests both similarities and differences in the patterns of association between time spent in physical activity, sleeping and in screen based activities and pain presence at 8 different body sites. In addition, they also suggest that the factors associated with the presence of pain, pain intensity and pain associated disability are different